Vehicle vision system with targetless camera calibration

ABSTRACT

A camera calibration system of a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. The camera is operable to capture image data. An image processor operable to process image data captured by the camera. The camera calibration system is operable to generate camera calibration parameters utilizing a bundle adjustment algorithm. Responsive to image processing of captured image data during movement of the vehicle along an arbitrary path, and responsive to the bundle adjustment algorithm, the camera calibration system is operable to calibrate the camera. The bundle adjustment algorithm may iteratively refine calibration parameters starting from a known initial estimation.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the filing benefits of U.S. provisionalapplications Ser. No. 61/878,877, filed Sep. 17, 2013, and Ser. No.61/825,752, filed May 21, 2013, which are hereby incorporated herein byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935; and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a collision avoidance system or visionsystem or imaging system for a vehicle that utilizes one or more cameras(such as one or more CMOS cameras) to capture image data representativeof images exterior of the vehicle, and provides a calibration systemthat is operable to determine calibration parameters for the camera orcameras of the vision system without use of fiducial markers or targetsin the field of view of the camera or cameras.

The cameras (such as one or more CMOS cameras) capture image datarepresentative of images exterior of the vehicle, and provide thecommunication/data signals, including camera data or captured imagedata, that may be displayed at a display screen that is viewable by thedriver of the vehicle, such as when the driver is backing up thevehicle, and that may be processed and, responsive to such imageprocessing, the system may detect an object at or near the vehicle andin the path of travel of the vehicle, such as when the vehicle isbacking up. The vision system may be operable to display a surround viewor bird's eye view of the environment at or around or at least partiallysurrounding the subject or equipped vehicle, and the displayed image mayinclude a displayed image representation of the subject vehicle.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a vision system thatincorporates cameras in accordance with the present invention;

FIG. 2 is a schematic showing the coordinate system and angles used torepresent the travel of the vehicle during calibration;

FIG. 3 is a perspective view and schematic of the vehicle;

FIG. 4 is a top level flow chart of the targetless software calibration(TSC) algorithm of the present invention;

FIG. 5 is a schematic showing the data acquisition and featureextraction of the system of the present invention;

FIGS. 6A and 6B are flowcharts of the targetless calibration algorithmof the present invention;

FIG. 7 shows details of a structure from motion algorithm or processsuitable for use with the system of the present invention;

FIG. 8 shows details of a bundle adjustment process of the system of thepresent invention;

FIG. 9 shows details of a feature detection algorithm or process of thesystem of the present invention;

FIG. 10 shows details of an extraction and matching of features processof the system of the present invention;

FIG. 11 shows details of a feature matching process of the system of thepresent invention;

FIG. 12 is a flow chart of the feature extraction and matching processor algorithm of the system of the present invention; and

FIG. 13 shows details of a feature filtering process of the system ofthe present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver assist system and/or objectdetection system and/or alert system operates to capture images exteriorof the vehicle and may process the captured image data to display imagesand to detect objects at or near the vehicle and in the predicted pathof the vehicle, such as to assist a driver of the vehicle in maneuveringthe vehicle in a rearward direction. The vision system includes an imageprocessor or image processing system that is operable to receive imagedata from one or more cameras and provide an output to a display devicefor displaying images representative of the captured image data.Optionally, the vision system may provide a top down or bird's eye orsurround view display and may provide a displayed image that isrepresentative of the subject vehicle, and optionally with the displayedimage being customized to at least partially correspond to the actualsubject vehicle.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes an imaging system or vision system 12that includes at least one exterior facing imaging sensor or camera,such as a rearward facing imaging sensor or camera 14 a (and the systemmay optionally include multiple exterior facing imaging sensors orcameras, such as a forwardly facing camera 14 b at the front (or at thewindshield) of the vehicle, and a sidewardly/rearwardly facing camera 14c, 14 d at respective sides of the vehicle), which captures imagesexterior of the vehicle, with the camera having a lens for focusingimages at or onto an imaging array or imaging plane or imager of thecamera (FIG. 1). The vision system 12 includes a control or electroniccontrol unit (ECU) or processor 18 that is operable to process imagedata captured by the cameras and may provide displayed images at adisplay device 16 for viewing by the driver of the vehicle (althoughshown in FIG. 1 as being part of or incorporated in or at an interiorrearview mirror assembly 20 of the vehicle, the control and/or thedisplay device may be disposed elsewhere at or in the vehicle). Thecameras operate to capture frames of image data at a desired or selectedframe rate, such as, for example, about 30 frames per second or more orless. The data transfer or signal communication from the camera to theECU may comprise any suitable data or communication link, such as avehicle network bus or the like of the equipped vehicle.

The present invention provides a targetless calibration system ortargetless software calibration (TSC) for calibrating cameras of avehicle vision system. The calibration system generates cameracalibration parameters, such as translational components: X, Y, Z (mm);and rotational components: Yaw, Pitch, and Roll angles (degrees). Nospecific vehicle motion required, and the calibration is performed asthe vehicle navigates along an arbitrary path. The bundle adjustment maybe a core feature of the targetless software calibration, anditeratively refines calibration parameters starting from a known initialestimation. The system uses bundle adjustment (BA) inputs, includingimage coordinates of matched scene features between two frames ofcaptured image data, vehicle motion vector (V=[S1, S2, ψ]) between twoframes of captured image data, an initial estimation of calibrationparameters, and/or three dimensional (3D) coordinates of features (suchas via a structure from motion algorithm or the like). The vehiclemotion vector (V) may be computed using only vehicle CAN Bus data, andmay be independent of any image data captured by the camera or camerasof the vehicle.

The system of the present invention thus may provide an algorithm thatperforms a fast and reliable on-line targetless calibration of one ormore cameras installed on a vehicle (in other words, the systemestimates camera calibration parameters). The system thus eliminates theneed for any camera calibration requiring special setup, such as, forexample, a vehicle manufacturer or factory end-of-line process orservice or the like.

The conditions for the calibration system include: R_(cal)=(Yaw, Pitch,Roll) and T_(cal)=(X_(cal), Y_(cal), Z_(cal)). The vehicle is moving onthe road with arbitrary patterns or texture (in other words, no lanemarkings or other markings or similar patterns or texturing needed) andalong an arbitrary path (with no special maneuvers being needed, such asplanned or specific turns or routing, such as driving in a circular loopor the like). The system may utilize a motion model for the cameracalibration, such as a kinematic model of vehicle motion of the typesdescribed in U.S. patent application Ser. No. 14/282,029, filed May 20,2014 by James Turk for VEHICLE VISION SYSTEM USING KINEMATIC MODEL OFVEHICLE MOTION, and published Nov. 27, 2014 as U.S. Publication No.US-2014-0350834 , and U.S. provisional application Ser. No. 61/825,753,filed May 21, 2013, which are hereby incorporated herein by reference intheir entireties.

Thus, and as can be seen with reference to FIGS. 5-8, the systemreceives image data from a camera or cameras of the vehicle and extractsand matches features over multiple images or frames of captured imagedata. The system also receives data from a kinematic model of vehiclemotion (such as a kinematic model of vehicle motion of the typesdescribed in U.S. patent application Ser. No. 14/282,029, filed May 20,2014 by James Turk for VEHICLE VISION SYSTEM USING KINEMATIC MODEL OFVEHICLE MOTION, and published Nov. 27, 2014 as U.S. Publication No.US-2014-0350834, and U.S. provisional application Ser. No. 61/825,753,filed May 21, 2013, which are hereby incorporated herein by reference intheir entireties), and determines the vehicle motion between frames and,based on initial parameter estimations and camera intrinsic parameters,the system determines the calibration parameters for the camera.

The targetless software calibration system of the present invention usessoftware modules such as an interface software module (which provides animage input or image data input and/or a network input or CAN input tothe system) and a kinematic model input. The system uses imageprocessing to provide feature detection, feature extraction, featurematching and feature filtering, and uses computer vision to provide acalibration parameters setup, structure-from-motion, a bundleadjustment, and an averaging of calibration results. For example, and asshown in FIG. 12, the system uses feature detection (FIG. 9), andextracts and matches feature descriptors (FIG. 10) and creates featurematches (FIG. 11), and outputs a list of valid features and theirmovements or movement patterns. The feature filtering (FIG. 13) enhancesthe structure-from-motion and bundle adjustment by eliminating featurepairs that do not satisfy the established rules for valid feature pairsin valid frame pairs.

As shown in FIG. 9, the feature detection may detect features within animage frame. The system thus may detect N features (such as threefeatures shown in FIG. 9) in three separate rectangular constant imageROIs, with the detection occurring periodically, such as every K frames(such as every frame or every other frame or every third frame or thelike). The feature detection may utilize various detection methods, suchas a Harris corner detector, speeded-up robust features (SURF), minimumeigenvalue and/or the like. The input may comprise a sequence of imageswith the ROI locations and dimensions, which may be constant for eachimage in the sequence. The output may comprise a list of pixelcoordinates for each detected feature in each image frame.

As shown in FIG. 10, the system may utilize extraction and matching offeature descriptors. For each detected feature in each image of thesequence of images, the system may build a descriptor, an m-dimensionalvector, based on the image information in the neighborhood of thefeature. The system may match the descriptor based on the distanced_(ij) from each other in frames i and j. The input may comprise imagecoordinates of the detected features in each image and the output maycomprise a descriptor for each features, a list of matched features anda match metric.

Thus, the present invention provides a targetless camera calibrationsystem that is operable to calibrate the cameras during any normaldriving movement of the vehicle equipped with the cameras and calibratedsystems. The system algorithm, when the system is calibrating more thanone camera of the vehicle, does not use overlap image areas between thecameras. The system may calibrate the cameras when the vehicle and/orcamera(s) undergo any movement in six degrees of freedom (such astranslational movements (x, y, z) forward/backward, side to side,up/down, and such as rotational movements pitch, yaw, roll). The systemachieves the calibration by matching features over multiple capturedimages and determining how the features position or location in thecaptured images may be different from where they would be with aproperly calibrated camera, based at least in part on an input of akinematic model of the actual vehicle movement.

Various vehicle camera calibration systems have been proposed, such asthose described in U.S. Pat. Nos. 8,421,865; 7,914,187; and/or7,949,486, and/or PCT Application No. PCT/US2011/036967, filed May 18,2011, and/or PCT Application No. PCT/CA2012/000378, filed Apr. 25, 2012,and/or PCT Application No. PCT/US2012/064980, filed Nov. 14, 2012, whichare all hereby incorporated herein by reference in their entireties.Such systems typically use and rely on a reference element on thevehicle itself. For outside viewing cameras (such as a side cameradisposed at or in an exterior rearview mirror of the vehicle), thevehicle manufacturer may initially calibrate the camera in the vehicleassembly plant, where the system is trained or set to correctly displayon the screen what is actually occurring or existing in real life on theground at or near the vehicle. If nothing changes over time, the systemwill continue to work properly. But over time, things change (forexample, the exterior mirror may be moved in any of six degrees offreedom, such as along the x, y and z directions and/or about threerotational axes), and the camera follows any changes in the mirror sothat what is shown on the display screen may no longer be true to theactual outside scene or reality. With multiple cameras, this may becomecritical if one camera is out of sync with other cameras. While use of areference point works to calibrate the cameras, the reference point mayalso be changed on the vehicle.

The system of the present invention calibrates the cameras of thevehicle multi-camera system without use of reference points on thevehicle. The system figures out that a camera is out of calibration andthen figures out how to calibrate it, all while the vehicle is beingnormally driven by the driver.

While the vehicle is driven along a road, the cameras capture frames ofimage data, and the system identifies or tags features in the capturedimages (such as, for example, a tree or a mailbox or a sign or the like)and then over a set of frames the system matches the features todetermine how they are moving in the captured image and relative to thevehicle movement. The system receives an input from a kinematic modelthat provides kinematic data that indicates exactly how the vehicle ismoving in reality. The control of the system thus knows how the vehiclemoved in reality (over the period of time that the frames were captured)and knows how the determined and matched features have moved as thecamera has seen them (over the period of time that the frames werecaptured), and thus the control sees how and where the features aremoving on the image plane itself. Thus, the system knows that for agiven real movement of the vehicle (such as based on a kinematic model),a determined feature in the camera's field of view should move in aparticular manner and should appear at a particular location at aparticular time or amount of movement of the vehicle (such as in asuccessive captured image or frame of image data). When the systemdetermines that a feature is not where it is expected to be, the systemcan determine that the camera is out of calibration or misaligned, andcan adjust the camera and/or image processing and/or displaycharacteristics accordingly.

The system may also determine if the object is also moving, in order toavoid a false determination of camera misalignment when the object ismoving as well as the vehicle and not with the vehicle (and thus doesnot appear where it is expected to be). For example, with a multi-camerasystem, two cameras may capture images of the object, and the system maydetermine whether or not the object is moving by comparing the imagedata captured by the two cameras. With the system determining that theobject is stationary (such as by seeing that the “movement” of theobject in the captured image data, such as due to the vehicle movement,is the generally the same between the two sets of multiple frames ofcaptured image data), the system can then determine whether one of thetwo cameras is misaligned, as discussed above.

Thus, the targetless calibration system of the present invention isoperable to calibrate a camera or cameras of a vehicle without use ofreference points on or off the vehicle. The output of the calibrationprocess is an exact or substantially exact location and orientation ofthe camera with respect to vehicle coordinate system (whereas knownsystems may calculate a transform function and use a correspondinglook-up table to generate a transfer map between the raw image data anddisplay screen). The system of the present invention has no constraintson the driving maneuvers of the vehicle (in other words, the driver isnot required to drive in a straight line for a period of time duringcalibration), and the system is operable to determine the calibration ofthe cameras during any driving maneuvers of the vehicle.

Also, if the vehicle is driven in straight line only, then only fourdegrees of freedom of the camera can be calibrated, namely the threerotational axes (pitch, yaw and roll) and the camera height. Suchstraight line driving constraints are unnatural constraints on driving.However, with the present invention, the vehicle may be driven in anormal manner, with turns involved. When the vehicle is drivennaturally, with turns involved, then all six degrees of freedom of thecamera can be checked and calibrated (so the location and orientation ofthe camera can be determined). The calibration time is shortened by thesystem of the present invention because no time is wasted when thevehicle turns naturally as the calibration process is not suspended forthe duration of the turn.

The camera or sensor may comprise any suitable camera or sensor.Optionally, the camera may comprise a “smart camera” that includes theimaging sensor array and associated circuitry and image processingcircuitry and electrical connectors and the like as part of a cameramodule, such as by utilizing aspects of the vision systems described inInternational Publication Nos. WO 2013/081984 and/or WO 2013/081985,which are hereby incorporated herein by reference in their entireties.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects orother vehicles or pedestrians or the like in the field of view of one ormore of the cameras. For example, the image processor may comprise anEyeQ2 or EyeQ3 image processing chip available from Mobileye VisionTechnologies Ltd. of Jerusalem, Israel, and may include object detectionsoftware (such as the types described in U.S. Pat. Nos. 7,855,755;7,720,580; and/or 7,038,577, which are hereby incorporated herein byreference in their entireties), and may analyze image data to detectvehicles and/or other objects. Responsive to such image processing, andwhen an object or other vehicle is detected, the system may generate analert to the driver of the vehicle and/or may generate an overlay at thedisplayed image to highlight or enhance display of the detected objector vehicle, in order to enhance the driver's awareness of the detectedobject or vehicle or hazardous condition during a driving maneuver ofthe equipped vehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ladar sensors or ultrasonicsensors or the like. The imaging sensor or camera may capture image datafor image processing and may comprise any suitable camera or sensingdevice, such as, for example, a two dimensional array of a plurality ofphotosensor elements arranged in at least 640 columns and 480 rows (atleast a 640×480 imaging array, such as a megapixel imaging array or thelike), with a respective lens focusing images onto respective portionsof the array. The photosensor array may comprise a plurality ofphotosensor elements arranged in a photosensor array having rows andcolumns. Preferably, the imaging array has at least 300,000 photosensorelements or pixels, more preferably at least 500,000 photosensorelements or pixels and more preferably at least 1 million photosensorelements or pixels. The imaging array may capture color image data, suchas via spectral filtering at the array, such as via an RGB (red, greenand blue) filter or via a red/red complement filter or such as via anRCC (red, clear, clear) filter or the like. The logic and controlcircuit of the imaging sensor may function in any known manner, and theimage processing and algorithmic processing may comprise any suitablemeans for processing the images and/or image data.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 7,005,974;5,760,962; 5,877,897; 5,796,094; 5,949,331; 6,222,447; 6,302,545;6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268;6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563;6,946,978; 7,859,565; 5,550,677; 5,670,935; 6,636,258; 7,145,519;7,161,616; 7,230,640; 7,248,283; 7,295,229; 7,301,466; 7,592,928;7,881,496; 7,720,580; 7,038,577; 6,882,287; 5,929,786 and/or 5,786,772,and/or International Publication Nos. WO 2011/028686; WO 2010/099416; WO2012/061567; WO 2012/068331; WO 2012/075250; WO 2012/103193; WO2012/0116043; WO 2012/0145313; WO 2012/0145501; WO 2012/145818; WO2012/145822; WO 2012/158167; WO 2012/075250; WO 2012/0116043; WO2012/0145501; WO 2012/154919; WO 2013/019707; WO 2013/016409; WO2013/019795; WO 2013/067083; WO 2013/070539; WO 2013/043661; WO2013/048994; WO 2013/063014, WO 2013/081984; WO 2013/081985; WO2013/074604; WO 2013/086249; WO 2013/103548; WO 2013/109869; WO2013/123161; WO 2013/126715; WO 2013/043661 and/or WO 2013/158592,and/or U.S. patent application Ser. No. 14/248,602, filed Apr. 9, 2014;Ser. No. 14/242,038, filed Apr. 1, 2014; Ser. No. 14/229,061, filed Mar.28, 2014; Ser. No. 14/343,937, filed Mar. 10, 2014; Ser. No. 14/343,936,filed Mar. 10, 2014; Ser. No. 14/195,135, filed Mar. 3, 2014; Ser. No.14/195,136, filed Mar. 3, 2014; Ser. No. 14/191,512, filed Feb. 27,2014; Ser. No. 14/183,613, filed Feb. 19, 2014; Ser. No. 14/169,329,filed Jan. 31, 2014; Ser. No. 14/169,328, filed Jan. 31, 2014; Ser. No.14/163,325, filed Jan. 24, 2014; Ser. No. 14/159,772, filed Jan. 21,2014; Ser. No. 14/107,624, filed Dec. 16, 2013; Ser. No. 14/102,981,filed Dec. 11, 2013; Ser. No. 14/102,980, filed Dec. 11, 2013; Ser. No.14/098,817, filed Dec. 6, 2013; Ser. No. 14/097,581, filed Dec. 5, 2013;Ser. No. 14/093,981, filed Dec. 2, 2013Ser. No. 14/093,980, filed Dec.2, 2013; Ser. No. 14/082,573, filed Nov. 18, 2013; Ser. No. 14/082,574,filed Nov. 18, 2013; Ser. No. 14/082,575, filed Nov. 18, 2013; Ser. No.14/082,577, filed Nov. 18, 2013; Ser. No. 14/071,086, filed Nov. 4,2013; Ser. No. 14/076,524, filed Nov. 11, 2013; Ser. No. 14/052,945,filed Oct. 14, 2013; Ser. No. 14/046,174, filed Oct. 4, 2013; Ser. No.14/016,790, filed Oct. 3, 2013; Ser. No. 14/036,723, filed Sep. 25,2013; Ser. No. 14/016,790, filed Sep. 3, 2013; Ser. No. 14/001,272,filed Aug. 23, 2013; Ser. No. 13/970,868, filed Aug. 20, 2013; Ser. No.13/964,134, filed Aug. 12, 2013; Ser. No. 13/942,758, filed Jul. 16,2013; Ser. No. 13/942,753, filed Jul. 16, 2013; Ser. No. 13/927,680,filed Jun. 26, 2013; Ser. No. 13/916,051, filed Jun. 12, 2013; Ser. No.13/894,870, filed May 15, 2013; Ser. No. 13/887,724, filed May 6, 2013;Ser. No. 13/852,190, filed Mar. 28, 2013; Ser. No. 13/851,378, filedMar. 27, 2013; Ser. No. 13/848,796, filed Mar. 22, 2012; Ser. No.13/847,815, filed Mar. 20, 2013; Ser. No. 13/800,697, filed Mar. 13,2013; Ser. No. 13/785,099, filed Mar. 5, 2013 ; Ser. No. 13/779,881,filed Feb. 28, 2013; Ser. No. 13/774,317, filed Feb. 22, 2013; Ser. No.13/774,315, filed Feb. 22, 2013; Ser. No. 13/681,963, filed Nov. 20,2012; Ser. No. 13/660,306, filed Oct. 25, 2012; Ser. No. 13/653,577,filed Oct. 17, 2012; and/or Ser. No. 13/534,657, filed Jun. 27, 2012,and/or U.S. provisional applications, Ser. No. 61/981,938, filed Apr.21, 2014; Ser. No. 61/981,937, filed Apr. 21, 2014; Ser. No. 61/977,941,filed Apr. 10, 2014; Ser. No. 61/977,940. filed Apr. 10, 2014; Ser. No.61/977,929, filed Apr. 10, 2014; Ser. No. 61/977,928, filed Apr. 10,2014; Ser. No. 61/973,922, filed Apr. 2, 2014; Ser. No. 61/972,708,filed Mar. 31, 2014; Ser. No. 61/972,707, filed Mar. 31, 2014; Ser. No.61/969,474, filed Mar. 24, 2014; Ser. No. 61/955,831, filed Mar. 20,2014; Ser. No. 61/953,970, filed Mar. 17, 2014; Ser. No. 61/952,335,filed Mar. 13, 2014; Ser. No. 61/952,334, filed Mar. 13, 2014; Ser. No.61/950,261, filed Mar. 10, 2014; Ser. No. 61/950,261, filed Mar. 10,2014; Ser. No. 61/947,638, filed Mar. 4, 2014; Ser. No. 61/947,053,filed Mar. 3, 2014; Ser. No. 61/941,568, filed Feb. 19, 2014; Ser. No.61/935,485, filed Feb. 4, 2014; Ser. No. 61/935,057, filed Feb. 3, 2014;Ser. No. 61/935,056, filed Feb. 3, 2014; Ser. No. 61/935,055, filed Feb.3, 2014; Ser. 61/931,811, filed Jan. 27, 2014; Ser. No. 61/919,129,filed Dec. 20, 2013; Ser. No. 61/919,130, filed Dec. 20, 2013; Ser. No.61/919,131, filed Dec. 20, 2013; Ser. No. 61/919,147, filed Dec. 20,2013; Ser. No. 61/919,138, filed Dec. 20, 2013, Ser. No. 61/919,133,filed Dec. 20, 2013; Ser. No. 61/918,290, filed Dec. 19, 2013; Ser. No.61/915,218, filed Dec. 12, 2013; Ser. No. 61/912,146, filed Dec. 5,2013; Ser. No. 61/911,666, filed Dec. 4, 2013; Ser. No. 61/911,665,filed Dec. 4, 2013; Ser. No. 61/905,461, filed Nov. 18, 2013; Ser. No.61/905,462, filed Nov. 18, 2013; Ser. No. 61/901,127, filed Nov. 7,2013; Ser. No. 61/895,610, filed Oct. 25, 2013; Ser. No. 61/895,609,filed Oct. 25, 2013; Ser. No. 61/879,837, filed Sep. 19, 2013; Ser. No.61/879,835, filed Sep. 19, 2013; Ser. No. 61/875,351, filed Sep. 9,2013; Ser. No. 61/869,195, filed. Aug. 23, 2013; Ser. No. 61/864,835,filed Aug. 12, 2013; Ser. No. 61/864,836, filed Aug. 12, 2013; Ser. No.61/864,837, filed Aug. 12, 2013; Ser. No. 61/864,838, filed Aug. 12,2013; Ser. No. 61/856,843, filed Jul. 22, 2013, Ser. No. 61/845,061,filed Jul. 11, 2013; Ser. No. 61/844,630, filed Jul. 10, 2013; Ser. No.61/844,173, filed Jul. 9, 2013; Ser. No. 61/844,171, filed Jul. 9, 2013;Ser. No. 61/842,644, filed Jul. 3, 2013; Ser. No. 61/840,542, filed Jun.28, 2013; Ser. No. 61/838,619, filed Jun. 24, 2013; Ser. No. 61/838,621,filed Jun. 24, 2013; Ser. No. 61/837,955, filed Jun. 21, 2013; Ser. No.61/836,900, filed Jun. 19, 2013; Ser. No. 61/836,380, filed Jun. 18,2013; Ser. No. 61/833,080, filed Jun. 10, 2013; Ser. No. 61/830,375,filed Jun. 3, 2013; Ser. No. 61/830,377, filed Jun. 3, 2013; Ser. No.61/823,648, filed May 15, 2013; Ser. No. 61/823,644, filed May 15, 2013;Ser. No. 61/821,922, filed May 10, 2013; Ser. No. 61/819,835, filed May6, 2013; Ser. No. 61/819,033, filed May 3, 2013; Ser. No. 61/816,956,filed Apr. 29, 2013; Ser. No. 61/815,044, filed Apr. 23, 2013; and/orSer. No. 61/813,361, filed Apr. 18, 2013, which are all herebyincorporated herein by reference in their entireties. The system maycommunicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in InternationalPublication Nos. WO/2010/144900; WO 2013/043661 and/or WO 2013/081985,and/or U.S. patent application Ser. No. 13/202,005, filed Aug. 17, 2011,which are hereby incorporated herein by reference in their entireties.

The imaging device and control and image processor and any associatedillumination source, if applicable, may comprise any suitablecomponents, and may utilize aspects of the cameras and vision systemsdescribed in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620; 5,670,935;5,796,094; 6,396,397; 6,806,452; 6,690,268; 7,005,974; 7,937,667;7,123,168; 7,004,606; 6,946,978; 7,038,577; 6,353,392; 6,320,176;6,313,454; and/or 6,824,281, and/or International Publication Nos. WO2010/099416; WO 2011/028686; and/or WO 2013/016409, and/or U.S. Pat.Publication No. US 2010-0020170, and/or U.S. patent application Ser. No.13/534,657, filed Jun. 27, 2012, which are all hereby incorporatedherein by reference in their entireties. The camera or cameras maycomprise any suitable cameras or imaging sensors or camera modules, andmay utilize aspects of the cameras or sensors described in U.S.Publication No. US-2009-0244361 and/or U.S. patent application Ser. No.13/260,400, filed Sep. 26, 2011, and/or U.S. Pat. Nos. 7,965,336 and/or7,480,149, which are hereby incorporated herein by reference in theirentireties. The imaging array sensor may comprise any suitable sensor,and may utilize various imaging sensors or imaging array sensors orcameras or the like, such as a CMOS imaging array sensor, a CCD sensoror other sensors or the like, such as the types described in U.S. Pat.Nos. 5,550,677; 5,670,935; 5,760,962; 5,715,093; 5,877,897; 6,922,292;6,757,109; 6,717,610; 6,590,719; 6,201,642; 6,498,620; 5,796,094;6,097,023; 6,320,176; 6,559,435; 6,831,261; 6,806,452; 6,396,397;6,822,563; 6,946,978; 7,339,149; 7,038,577; 7,004,606; 7,720,580; and/or7,965,336, and/or International Publication Nos. WO/2009/036176 and/orWO/2009/046268, which are all hereby incorporated herein by reference intheir entireties.

The camera module and circuit chip or board and imaging sensor may beimplemented and operated in connection with various vehicularvision-based systems, and/or may be operable utilizing the principles ofsuch other vehicular systems, such as a vehicle headlamp control system,such as the type disclosed in U.S. Pat. Nos. 5,796,094; 6,097,023;6,320,176; 6,559,435; 6,831,261; 7,004,606; 7,339,149; and/or 7,526,103,which are all hereby incorporated herein by reference in theirentireties, a rain sensor, such as the types disclosed in commonlyassigned U.S. Pat. Nos. 6,353,392; 6,313,454; 6,320,176; and/or7,480,149, which are hereby incorporated herein by reference in theirentireties, a vehicle vision system, such as a forwardly, sidewardly orrearwardly directed vehicle vision system utilizing principles disclosedin U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,877,897; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; and/or 7,859,565, which are all herebyincorporated herein by reference in their entireties, a trailer hitchingaid or tow check system, such as the type disclosed in U.S. Pat. No.7,005,974, which is hereby incorporated herein by reference in itsentirety, a reverse or sideward imaging system, such as for a lanechange assistance system or lane departure warning system or for a blindspot or object detection system, such as imaging or detection systems ofthe types disclosed in U.S. Pat. Nos. 7,881,496; 7,720,580; 7,038,577;5,929,786 and/or 5,786,772, and/or U.S. provisional applications, Ser.No. 60/628,709, filed Nov. 17, 2004; Ser. No. 60/614,644, filed Sep. 30,2004; Ser. No. 60/618,686, filed Oct. 14, 2004; Ser. No. 60/638,687,filed Dec. 23, 2004, which are hereby incorporated herein by referencein their entireties, a video device for internal cabin surveillanceand/or video telephone function, such as disclosed in U.S. Pat. Nos.5,760,962; 5,877,897; 6,690,268; and/or 7,370,983, and/or U.S.Publication No. US-2006-0050018, which are hereby incorporated herein byreference in their entireties, a traffic sign recognition system, asystem for determining a distance to a leading or trailing vehicle orobject, such as a system utilizing the principles disclosed in U.S. Pat.Nos. 6,396,397 and/or 7,123,168, which are hereby incorporated herein byreference in their entireties, and/or the like.

Optionally, the circuit board or chip may include circuitry for theimaging array sensor and or other electronic accessories or features,such as by utilizing compass-on-a-chip or EC driver-on-a-chip technologyand aspects such as described in U.S. Pat. No. 7,255,451 and/or U.S.Pat. No. 7,480,149; and/or U.S. Publication No. US-2006-0061008 and/orU.S. patent application Ser. No. 12/578,732, filed Oct. 14, 2009, whichare hereby incorporated herein by reference in their entireties.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device disposed at or in the interior rearview mirror assemblyof the vehicle, such as by utilizing aspects of the video mirror displaysystems described in U.S. Pat. No. 6,690,268 and/or U.S. patentapplication Ser. No. 13/333,337, filed Dec. 21, 2011, which are herebyincorporated herein by reference in their entireties. The video mirrordisplay may comprise any suitable devices and systems and optionally mayutilize aspects of the compass display systems described in U.S. Pat.Nos. 7,370,983; 7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593;4,546,551; 5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851;5,708,410; 5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508;6,222,460; 6,513,252; and/or 6,642,851, and/or European patentapplication, published Oct. 11, 2000 under Publication No. EP 0 1043566,and/or U.S. Publication No. US-2006-0061008, which are all herebyincorporated herein by reference in their entireties. Optionally, thevideo mirror display screen or device may be operable to display imagescaptured by a rearward viewing camera of the vehicle during a reversingmaneuver of the vehicle (such as responsive to the vehicle gear actuatorbeing placed in a reverse gear position or the like) to assist thedriver in backing up the vehicle, and optionally may be operable todisplay the compass heading or directional heading character or iconwhen the vehicle is not undertaking a reversing maneuver, such as whenthe vehicle is being driven in a forward direction along a road (such asby utilizing aspects of the display system described in InternationalPublication No. WO 2012/051500, which is hereby incorporated herein byreference in its entirety).

Optionally, the vision system (utilizing the forward facing camera and arearward facing camera and other cameras disposed at the vehicle withexterior fields of view) may be part of or may provide a display of atop-down view or birds-eye view system of the vehicle or a surround viewat the vehicle, such as by utilizing aspects of the vision systemsdescribed in International Publication Nos. WO 2010/099416; WO2011/028686; WO2012/075250; WO 2013/019795; WO 2012/075250; WO2012/145822; WO 2013/081985; WO 2013/086249; and/or WO 2013/109869,and/or U.S. patent application Ser. No. 13/333,337, filed Dec. 21, 2011,which are hereby incorporated herein by reference in their entireties.

Optionally, a video mirror display may be disposed rearward of andbehind the reflective element assembly and may comprise a display suchas the types disclosed in U.S. Pat. Nos. 5,530,240; 6,329,925;7,855,755; 7,626,749; 7,581,859; 7,446,650; 7,370,983; 7,338,177;7,274,501; 7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187 and/or6,690,268, and/or in U.S. Publication Nos. US-2006-0061008 and/orUS-2006-0050018, which are all hereby incorporated herein by referencein their entireties. The display is viewable through the reflectiveelement when the display is activated to display information. Thedisplay element may be any type of display element, such as a vacuumfluorescent (VF) display element, a light emitting diode (LED) displayelement, such as an organic light emitting diode (OLED) or an inorganiclight emitting diode, an electroluminescent (EL) display element, aliquid crystal display (LCD) element, a video screen display element orbacklit thin film transistor (TFT) display element or the like, and maybe operable to display various information (as discrete characters,icons or the like, or in a multi-pixel manner) to the driver of thevehicle, such as passenger side inflatable restraint (PSIR) information,tire pressure status, and/or the like. The mirror assembly and/ordisplay may utilize aspects described in U.S. Pat. Nos. 7,184,190;7,255,451; 7,446,924 and/or 7,338,177, which are all hereby incorporatedherein by reference in their entireties. The thicknesses and materialsof the coatings on the substrates of the reflective element may beselected to provide a desired color or tint to the mirror reflectiveelement, such as a blue colored reflector, such as is known in the artand such as described in U.S. Pat. Nos. 5,910,854; 6,420,036; and/or7,274,501, which are hereby incorporated herein by reference in theirentireties.

Optionally, the display or displays and any associated user inputs maybe associated with various accessories or systems, such as, for example,a tire pressure monitoring system or a passenger air bag status or agarage door opening system or a telematics system or any other accessoryor system of the mirror assembly or of the vehicle or of an accessorymodule or console of the vehicle, such as an accessory module or consoleof the types described in U.S. Pat. Nos. 7,289,037; 6,877,888;6,824,281; 6,690,268; 6,672,744; 6,386,742; and/or 6,124,886, and/orU.S. Publication No. US-2006-0050018, which are hereby incorporatedherein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The invention claimed is:
 1. A camera calibration system of a vehicle,said camera calibration system comprising: a camera disposed at avehicle and having a field of view exterior of the vehicle; wherein saidcamera comprises a two dimensional array having a plurality ofphotosensing elements; wherein said camera is operable to capture imagedata; an image processor operable to process image data captured by saidcamera; wherein said camera calibration system is operable to generatecamera calibration parameters utilizing a bundle adjustment algorithm;wherein said bundle adjustment algorithm uses a kinematic model ofvehicle motion derived from image processing by said image processor ofmultiple frames of captured image data; wherein said bundle adjustmentalgorithm generates calibration parameters utilizing the equation:X _(image) =P*X _(w), where X_(image) is a selection of image points incaptured image data, P is a projective matrix representing motionbetween frames of captured image data, and X_(w) is a vector of worldcoordinates of image points in captured image data; wherein, responsiveto image processing by said image processor of multiple frames ofcaptured image data, said camera calibration system extracts and matchesfeatures determined in multiple frames of captured image data; wherein,responsive to image processing by said image processor of captured imagedata during movement of the vehicle along an arbitrary path, andresponsive to generation of camera calibration parameters utilizing saidbundle adjustment algorithm, said camera calibration system is operableto calibrate said camera; and wherein said camera calibration systemcalibrates said camera as the vehicle navigates the arbitrary path, andwherein calibration of said camera includes correction for misalignmentin yaw, pitch and roll angles of said camera.
 2. The camera calibrationsystem of claim 1, wherein said bundle adjustment algorithm iterativelyrefines calibration parameters starting from a known initial estimation.3. The camera calibration system of claim 1, wherein said bundleadjustment algorithm solves for calibration parameters and actualcoordinates of an object present in the field of view of said camera. 4.The camera calibration system of claim 1, wherein said image processoris operable to calibrate said camera without image processing of imagedata representative of captured images of a target.
 5. The cameracalibration system of claim 1, comprising a plurality of camerasdisposed at the vehicle and having respective fields of view exterior ofthe vehicle, wherein said image processor is operable to process imagedata captured by said cameras, and wherein said image processor isoperable to calibrate said cameras without use of targets in the fieldof view of said cameras and without image processing of image datarepresentative of captured images of targets.
 6. The camera calibrationsystem of claim 1, wherein the generated camera calibration parameterscomprise at least one of (i) at least one of translational components X,Y, Z and (ii) at least one of rotational components Yaw, Pitch and Roll.7. The camera calibration system of claim 1, wherein the generatedcamera calibration parameters comprise (i) translational components X,Y, Z and (ii) rotational components Yaw, Pitch and Roll angles.
 8. Thecamera calibration system of claim 1, wherein said camera calibrationsystem comprises a plurality of cameras disposed at the vehicle, eachhaving a respective field of view exterior of the vehicle.
 9. The cameracalibration system of claim 8, wherein image data captured by at leastsome of said plurality of cameras is used for a surround view system ofthe vehicle.
 10. The camera calibration system of claim 9, wherein oneof said cameras comprises a rear camera disposed at a rear portion ofthe vehicle, and another of said cameras comprises a side cameradisposed at a driver-side exterior rearview mirror assembly of thevehicle and another of said cameras comprises a passenger side cameradisposed at a passenger-side exterior rearview mirror assembly of thevehicle.
 11. A camera calibration system of a vehicle, said cameracalibration system comprising: a plurality of cameras disposed at avehicle, each having a respective field of view exterior of the vehicle;wherein said plurality of cameras comprises (i) a rear camera disposedat a rear portion of the vehicle and having a rearward field of view,(ii) a driver-side camera disposed at a driver-side exterior rearviewmirror assembly of the vehicle and at least having a generally sidewardand rearward field of view and (iii) a passenger-side camera disposed ata passenger-side exterior rearview mirror assembly of the vehicle and atleast having a generally sideward and rearward field of view; whereinsaid cameras are operable to capture image data; an image processoroperable to process image data captured by said cameras; wherein saidcamera calibration system is operable to generate camera calibrationparameters for each of said cameras utilizing a bundle adjustmentalgorithm; wherein said bundle adjustment algorithm uses a kinematicmodel of vehicle motion derived from image processing by said imageprocessor of multiple frames of image data captured by each of saidcameras; wherein said bundle adjustment algorithm generates calibrationparameters utilizing the equation:X _(image) =P*X _(w), where X_(image) is a selection of image points incaptured image data, P is a projective matrix representing motionbetween frames of captured image data, and X_(w) is a vector of worldcoordinates of image points in captured image data; wherein, responsiveto image processing by said image processor of multiple frames of imagedata captured by each of said cameras, said camera calibration systemextracts and matches features determined in multiple frames of imagedata captured by each of said cameras; wherein, responsive to imageprocessing by said image processor of captured image data duringmovement of the vehicle along an arbitrary path, and responsive togeneration of camera calibration parameters utilizing said bundleadjustment algorithm, said camera calibration system is operable tocalibrate each of said cameras; and wherein said camera calibrationsystem calibrates each of said cameras as the vehicle navigates thearbitrary path, and wherein calibration of each of said cameras includescorrection for misalignment in yaw, pitch and roll angles of each ofsaid cameras.
 12. The camera calibration system of claim 11, whereinsaid bundle adjustment algorithm iteratively refines calibrationparameters starting from a known initial estimation.
 13. The cameracalibration system of claim 11, wherein said bundle adjustment algorithmsolves for calibration parameters and actual coordinates of an objectpresent in the field of view of said camera.
 14. The camera calibrationsystem of claim 11, wherein the generated camera calibration parameterscomprise at least one of (i) at least one of translational components X,Y, Z and (ii) at least one of rotational components Yaw, Pitch and Roll.15. The camera calibration system of claim 11, wherein the generatedcamera calibration parameters comprise (i) translational components X,Y, Z and (ii) rotational components Yaw, Pitch and Roll.
 16. The cameracalibration system of claim 11, wherein each of said cameras comprises atwo dimensional array of a plurality of photosensing elements.
 17. Thecamera calibration system of claim 11, wherein image data captured by atleast some of said plurality of cameras is used for a surround viewsystem of the vehicle.
 18. A camera calibration system of a vehicle,said camera calibration system comprising: a plurality of camerasdisposed at the vehicle, each having a respective field of view exteriorof the vehicle; wherein each of said cameras comprises a two dimensionalarray of a plurality of photosensing elements; wherein said plurality ofcameras comprises a rear camera disposed at a rear portion of thevehicle, a driver side camera disposed at a driver-side exteriorrearview mirror assembly of the vehicle and a passenger side cameradisposed at a passenger-side exterior rearview mirror assembly of thevehicle; wherein each of said cameras is operable to capture image data;an image processor operable to process image data captured by saidcameras; wherein image data captured by at least some of said pluralityof cameras is used for a surround view system of the vehicle; whereinsaid camera calibration system is operable to generate cameracalibration parameters utilizing a bundle adjustment algorithm; whereinsaid bundle adjustment algorithm uses a kinematic model of vehiclemotion derived from image processing by said image processor of multipleframes of image data captured by each of said cameras; wherein saidbundle adjustment algorithm generates calibration parameters utilizingthe equation:X _(image) =P*X _(w), where X_(image) is a selection of image points incaptured image data, P is a protective matrix representing motionbetween frames of captured image data, and X_(w) is a vector of worldcoordinates of image points in captured image data; wherein, responsiveto image processing by said image processor of multiple frames of imagedata captured by each of said cameras, said camera calibration systemextracts and matches features determined in multiple frames of imagedata captured by each of said cameras; wherein the generated cameracalibration parameters comprise (i) at least one of translationalcomponents X, Y, Z and (ii) at least one of rotational components Yaw,Pitch and Roll; wherein, responsive to image processing by said imageprocessor of captured image data during movement of the vehicle along anarbitrary path, and responsive to generation of camera calibrationparameters utilizing said bundle adjustment algorithm, said cameracalibration system is operable to calibrate at least one of saidcameras; and wherein said camera calibration system calibrates each ofsaid cameras as the vehicle navigates the arbitrary path, and whereincalibration of each of said cameras includes correction for misalignmentin yaw, pitch and roll angles of each of said cameras.
 19. The cameracalibration system of claim 18, wherein said bundle adjustment algorithmiteratively refines calibration parameters starting from a known initialestimation.
 20. The camera calibration system of claim 19, wherein saidbundle adjustment algorithm solves for calibration parameters and actualcoordinates of an object present in the field of view of said at leastone of said cameras.